Supporting On-the-fly Provenance Tracking in Stream Processing Systems

A new class of data management systems that operate on highvolume streaming data is becoming increasingly important. As this kind of systems has to process unpredictable streaming data in real-time and deliver instantaneous responses, it becomes very difficult to precisely validate stream processing results in timely manner, verify stream computation that took place and investigate processing steps used to generate result data. Therefore, a mechanism that can precisely track provenance of data streams at execution time is crucial for confidence in the results produced by this kind of systems. This paper presents a novel on-the-fly stream provenance tracking mechanism that enables a collection of provenance queries to be performed dynamically without requiring provenance information to be stored persistently. The experimental results indicate that the impact of provenance collection on system performance is relatively small (7% overhead observed). In addition, our provenance solution offers low-latency processing (about 0.3 ms per additional component) with reasonable memory consumption.